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Factors Influencing Mail Survey Response Rates: 

What Do We Really Know? 

According to Dillman (1991), the predominant sample survey method used in this country is the 
mail survey. As such, it has been the subject of a considerable body of research. Most studies of mail survey 
methodology focus on specific design elements, but several major reviews of research that attempt to 
synthesize the findings of the independent empirical studies and to identify the techniques that reliably 
stimulate responses in mail surveys have been published. In addition, other less comprehensive reviews 
focusing on specific survey design variables have been published. The methods used by the researchers in 
these reviews have varied, and not all of the more recent studies have utilized quantitative analytical 
procedures. The relative merit of the findings are somewhat dependent on the way in which the reviews 
were conducted and reported, of course, but both types of reviews can provide valuable information for other 
researchers in this field regarding mail survey procedures and potential publication avenues for research on 
mail surveys. 

Reviews of research are expected to preface new research studies but are also soni? times conducted 
on a more extensive scale as independent endeavors. Much less attention has been placed on methods for 
conducting and reporting reviews of previous research studies than on new or primary research studies. 
Jackson (1980) focused on integrative reviews of research, documenting and evaluating various methods by 
which they had been accomplished. The use of meta-analysis provided a major change in comparing the 
results of multiple studies. Cooper (1989) noted that the reviewing of previous research studies had been 
facilitated by the recent availability of computerized literature searches and the development of 
quantitative procedures for analysis. 

The purpose of the present study was to examine the review articles themselves, identifying 
procedures and potential publication sources for use by those interested in the study of mail survey methods. 
For purposes of this study, review articles are limited to those that are comprehensive in nature and focus 
on identifying techniques that facilitate response rates in mail surveys. In the present context, 
"comprehensive" is defined as including several survey design variables that have possible impacts on 
survey response rates, and "source studies" are the citations (published or unpt blished, including books) 
referenced by the review author(s) that were included in their quantitative analyses or that contributed to 
the formulation of their conclusions in qualitative reviews. 

Method 

Procedures for Identifying Review Articles 

The first step in identifying review articles was to search computerized CD-ROM databases 
containing abstracts in the fields of psychology, sociology, business and marketing, and education. The 

The authors extend their appreciation to Dr. Kathy Green, Denver University, for her review of this paper 
and her helpful comments. 



databases used were PSYCLit (1974-1992), Sociofile (1974-1992), ABl/Inform (1987-1992), and ERIC (1966- 
1992). Yu and Cooper (1983) had used "response rate" and "survey 7 ' in searching for articles for their 
review. Yammarino, Skinner, and Childers (1991) used "response rates" and "mail surveys" as key words. 
An initial list of abstracts from each database was compiled by/ijsing the key words "response rate" or 
"response rates" and "mail survey" or "mail surveys." The ABI/Inform database searched only for the 
singular forms, amending the plural to the singular for both "rates" and "surveys." Jackson (1980), in 
searching for review articles, had looked for articles under the headings of "literature reviews" and 
"research reviews." Further defining the scope of the present search by adding the term "research review" 
or "literature review" was too limiting, however, sometimes producing no citations, so the more inclusive 
initial lists were used. Reference lists in the review articles identified through the computer searches were 
examined for other review articles not included in the computer databases. 

Four acceptable review articles were found in PSYCLit among the 109 articles identified as 
containing "response rate” or "response rates" and "mail survey" or "mail surveys." Four review articles 
were among the 74 abstracts listed from ABI/Inform, and two from the 66 drawn from Sociofile. There were 
no review articles that met the criteria for this study among the 137 abstracts produced by the ERIC 
searches. Some of the acceptable review articles appeared in more than one database. One review article 
was indexed in three of the databases, two were found in both PSYCLit and ABI/Inform. When duplication 
was eliminated, there were six review articles that met the criteria for inclusion in the present study. 
Examination of the reference lists from those articles produced three additional reviews for the total of 
nine that serve as the basis for this study (see References: Reviews Included in the Study). 

Selection of Reviews 

Some of the review articles initially identified were not included in this study because they 
focused solely on a specific aspect of survey design, such as monetary incentives (Armstrong, 1975), cover 
letter personalization (Worthen St Valcarce, 1985), or type of postage used (Armstrong St Lusk, 1987). One 
review (Yu & Cooper, 1983) that was not included was not limited to mail surveys, and it did not 
distinguish between mail surveys and other types of surveys in the presentation of many of the findings. 
Berdie, Anderson, and Niebuhr (1986) also went beyond mail surveys. Houston and Ford's (1976) review 
examined response speed and quality, rather than response rate, and was excluded from the present study. 
Reviews that did not contain or make available a list of the source articles, such as Goyder (1982), were 
also not included. Dillman (1991) cited specific studies to illustrate the points he made regarding specific 
mail survey techniques as part of a comprehensive system for conducting a survey. Jobber and Saunders 
(1993) limited their review and model development to mail surveys of commercial populations. 

Variables 

For each review article, the following information was sought: date and journal in which the 
review article appeared, type of analysis used, criteria for inclusion of source studies, findings, number of 
source studies, years and journals in which source studies were published, method by which source studies 
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were identified. When discrepancies appeared between reference lists in the listings of source articles, the 
original articles were obtained to determine the correct reference listing. 

Source Studies 

Some review articles (Bruvold & Comer, 1988; Fox, Crask, ic Kim, 1988; Heberlein & Baumgartner, 
1978; Yammarino, Skinner, & Childers, 1991) provided reference lists of the articles used in the 
quantitative analysis (source studies) and separate lists of references providing background or introductory 
material. This clearly identified the source studies, although the titles of some of the source studies did 
not clearly delineate them as relevant for mail surveys. In Conant, Smart, and Walker (1990), the review 
of articles to determine response facilitation techniques was only the first stage in a more involved study. 
The relevant citations were included in a single table, making them easy to discern. 

For the other four reviews, the two researchers independently read the articles and determined 
which sources were background or general references and which should be included as source studies. They 
then compared their lists and resolved discrepancies through discussion and clarification of the criteria for 
inclusion. It was decided that references that were cited as leading to the conclusions of the review author 
should be included, even though they were unpublished works, books, or articles that appeared to be 
inappropriate because their titles indicated a focus on surveys other than those conducted through the 
mail. 

A total of 329 source studies was used in the nine review articles. Some source articles were 
referenced incorrectly, initially causing them to be considered additional articles. Differences (errors) 
occurred in publication year, journal pages and/or volume number, journal title, author, spelling of author's 
name, and even title of the article. 

The following information for each source study was entered into a Fox Base/MAC database: 
author(s), year of publication, journal in which it was published, and which review articles had used the 
study. Citation as a reference in each of the nine review articles was coded as a dichotomous variable. 
Because of the small number of review articles and the varying procedures used in them, frequency 
distributions were used in analyzing the data in addition to descriptions of the reviewers' procedures. 

‘ Results 

Publication of Review Articles 

Three of the nine reviews were published in Public Opinion Quarterly, three in marketing research 
journals (Journal of the Market Research Society and Journal of Marketing Research), and one each in the 
Journal of Business Research, Journal of Management, and the American Sociological Review (see Table 1). 
None of the reviews appeared in either education or psychological journals. 

Procedures Used in Review Articles 

Four of the reviews utilized quantitative analyses, while the other five could best be described as 
qualitative. In general, the quantitative review articles contained better documentation of the methods 
used than the qualitative reviews. For example, none of the five qualitative reviews contained 
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information about how the source articles were identified or the criteria for inclusion. The four 
quantitative reviews, in contrast, were detailed in describing methods for locating the articles, selection 
criteria, and analysis. 

Locating Source Studies 

Heberlein and Baumgartner (1978), the earliest quantitative review, did not use computerized 
searches, as did authors of the later quantitative reviews. Computerized search services were relatively 
new at that time. Heberlein and Baumgartner published their review in a sociology journal. The Sociofile 
indexing system in use today (which indexes journals in the field of sociology) only dates back to 1974, as 
does PSYCLit. Social SciSearch began in 1972. The primary source of articles for the Heberlein and 
Baumgartner review was an annotated bibliography by Potter, Sharp, Hendee, and dark (1972) containing 
193 citations, supplemented by manual reviews of journals and citations in the published articles that were 
located. The review authors did not, however, identify the journals which were searched manually. 



Table 1 



Review 

Article 


Pub. 

Year 


Journal 


No. Source 
Articles 


Analysis 


Yammarino, Skinner, 












& Childers 


1991 


Public Opinion Quarterly 




115 


meta-analysis 


Conant, Smart, & Walker 1990 


Journal of the Market Research Soc. 


52 


qualitative 


Fox, Crask, & Kim 


1988 


Public Opinion Quarterly 




40 


meta-analysis 


Bruvold & Comer 


1988 


Journal of Business Research 




212 


weighted-least-squares 












logit regression 


Harvey 


1987 


Journal of the Market Research Soc. 


129 


qualitative 


Duncan 


1979 


Journal of Management 




60 


qualitative 


Heberlein & Baumgartner 1978 


American Sociological Review 




95 


quantitative 


Kanuk & Berenson 


1975 


Journal of Marketing Research 




69 


qualitative 


Linsky 


1975 


Public Opinion Quarterly 




57 


qualitative 



Bruvold and Comer (1988), began with the list of studies used by Heberlein and Baumgartner. This 
was supplemented by manual searches of five journals (Journal of Marketing Research, Journal of Marketing, 
Public Opinion Quarterly, Journal of Business Research, and Journal of Applied Psychology) for the years 
1964-1980. The reviewers also conducted computer searches on six databases: Social SciSearch, 
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PsychINFO, ABI/ Inform, Sociological Abstracts, U.S. Political Science Documents, and ERIC. For key 
words, they used "response rate" and "mail survey or mail questionnaire." They included only published 
journal articles and one reference published in a conference proceedings. 

Fox, Crask, and Kim (1988) conducted a manual search of three of the same journals as Bruvold and 
Comer (Journal of Marketing Research, Public Opinion Quarterly, and Journal of Applied Psychology) for 
the previous 25 years or, in the case of the Journal of Marketing Research, since 1964 when it began 
publication. They also did a computerized search using PsychINFO and examining the reference lists from 
relevant articles. The keywords used in the computer search were not identified. The reviewers used only 
published articles that were experimental studies. Response rate variables that were coded subjectively, 
such as "salience" that had been included in the Heberlein and Baumgartner (1978) study, were not 
considered. 

Yammarino, Skinner, and Childers (1991) ’ »n by examining reference lists of previously 
published review articles, including Kanuk and Berenson (1975), Linsky (1975), Heberlein and Baumgartner 
(1978), Yu and Cooper (1983), Harvey (1987), and Fox, Crask, and Kim (1988). They then conducted 
computer searches of the following databases: ABI/Inform, ERIC, Social SciSearch, Sociological Abstracts, 
and PsydNFO. The keywords of "mail surveys," "response rates," were used and also combined with the 
names and synonyms for the 17 response rate variables targeted for study. To identify recent studies (1978 
and later), a manual search of each of the following ten journals was performed: American Sociological 
Review, Journal of the Academy of Marketing Science, Journal of Advertising Research, Journal of the 
American Statistical Association, Journal of Applied Psychology, Journal of Marketing, Journal of 
Marketing Research, Journal of the Market Research Society, Psychological Reports, and Public Opinion 
Quarterly. To be included, the source studies had to have a manipulated factor and had to include response 
rates for the various conditions. 

Selection Criteria Differences 

Six of the reviews limited their source citations to published articles (Duncan, 1979; Fox, Crask, & 
Kim, 1988) and a published book (Kanuk & Berenson, 1975) or proceedings (Bruvold & Comer, 1988; Conant, 
Smart, & Walker, 1990; Heberlein & Baumgartner, 1978). The other three reviews (Linsky, 1975; Harvey, 
1987; Yammarino, Skinner, & Childers, 1991) included one or more unpublished papers and, in the 
Yammarino, Skinner, and Childers review, five unpublished studies cited in a Public Opinion Quarterly 
review by Armstrong and Lusk (1987). 

Each of the eight books was cited in only a single review. With one exception, the unpublished 
papers were also unique to a single review, as were the five unpublished studies cited by Armstrong and 
Lusk (1987). There was only one instance of duplication among the studies cited from meeting proceedings. 

Three journals accounted for almost half of the source studies (see Table 2) . One fourth of the source 
studies had been published in Public Opinion Quarterly. The Journal of Marketing Research and the Journal 
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Table 2 



Publication Sources of Studies Cited in Reviews 



Sourc e Studies 

Journal Number Per Cent 



Public Opinion Quarterly 


83 


25.2 


Journal of Marketing Research 


41 


12.5 


Journal of Applied Psychology 


33 


10.0 


Journal of Marketing 


14 


4.3 


Journal of Advertising Research 


14 


4.3 


American Sociological Review 


13 


4.0 


Journal of the Market Research Society 


10 


3.0 


Journal of the Academy of Marketing Science 


7 


2.1 


Journal of the American Statistical Association 


7 


2.1 


Psychological Reports 


6 


1.8 


Sociology and Social Research 


5 


1.5 


Journal of Business Research 


4 


1.2 


Journal of Personality and Social Psychology 


4 


1.2 


Printers Ink 


3 


0.9 


American Marketing Association Educators' Conference Proceedings 


3 


0.9 


Journals publishing two source articles (10 journals) 


20 


6.1 


Journals publishing one source article (44 journals) 


44 


13.4 


Books 


8 


2.4 


Unpublished manuscripts 


5 


15 


Studies cited in previous publication but not published independently 


5 


15 



of Applied Psychology accounted for another 225 percent of the studies. Fifty-four journals included had 
published no more than two of the dted source studies. 

Analytic Procedures 

Heberlein and Baumgartner (1978' based their analysis on procedures introduced by Sudman and 
Bradbum (1974) and a similar procedure by Glass and Smith (1976), one of the early works on meta- 
analysis. Bruvold and Comer (1988) endeavored to develop a model to estimate mail survey response rates 
using a weighted-least squares regression procedure based on Berkson (1944), Flath and Leonard (1979), and 
Green, Carmone, and Wachspress (1977). Fox, Crask and Kim (1988) labeled their analysis as meta- 
analysis and described it as similar to that used by Armstrong and Lusk (1987) and Yu and Cooper (1983). 
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Yammarino, Skinner and Childers (1991) also used meta-analytic procedures, citing Glass (1977), Hunter 
and Schmidt (1990), and Hunter, Schmidt, and Jackson (1982) as methodological references. 

Presentation of Findings in Qualitative Reviews 

Linsky (1975), Kanuk and Berenson (1975) and Duncan (1979) utilized tables in summarizing the 
impact of specific manipulations. Conant, Smart, and Walker (1990) presented a single table summarizing 
the effects of the various techniques. Harvey (1987) used only text descriptions. None of the three articles 
appearing in the marketing research journals used quantitative procedures. 

Findings From the Reviews 

Four of the reviews contain no descriptions of the methods used by the authors in selecting the 
studies, so the discussion of findings will limited to the remaining five review articles: Bruvold and Comer 
(1988); Fox, Crask and Kim (1988); Heberlein and Baumgartner (1978); Linsky (1975); and Yammarino, 
Skinner and Childers (1991). Not all of the review authors chose the same variables to study (see Figures 
1, 2, and 3), nor were the variables necessarily defined in the same way. The discussion of findings will 
have to take into account both the variables and their definitions. It is important to note the variables 
that were found not significantly related to or predictive of response rate, as well as those that were. 

Fox, Crask and Kim (1988) limited their investigations to 10 variables, Yammarino, Skinner and 
Childers (1991) to 17. Although Linsky (1975) discussed a fairly large number of variables, some had been 
the subject of only one or two research studies. His summary comments targeted 10 aspects of mail surveys 
although findings regarding some of them were inconclusive. Heberlein and Baumgartner (1978) coded 71 
variables but used only 10 of them in their regression model to predict response rates. Bruvold and Comer 
(1988) investigated (coded) a fairly large number of variables and derived coefficients or weights for most 
of them through regression. ^ Most variables used in the latter two reviews were coded 1/0 (present or 
absent). Also, in meta-analysis, some categories were collapsed to attain sufficient cases for analysis, thus 
losing some degree of specificity. 

Another area that deserves attention is the number of studies that contributed to the conclusions in 
the review articles. The number of research studies has increased considerably since Linsky's review in 
1975. Also, the review articles cannot be considered independent because many, if not most, of the research 
studies on which they are based are cited in more than one of the reviews. 

Heberlein and Baumgartner (1978), after conducting their initial analyses, proceeded to additional 
analyses in which the effects of contacts and salience were controlled. Yammarino, Skinner and Childers 

1 In the following discussions, effects in the quantitative reviews will be reported as the percentage of 
increase or decrease in response rate that can be expected due to the use of the various procedures, although 
those estimates were not necessarily derived in the same manner. Readers are encouraged to consult the 
review articles for further details. 
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(1991) completed their initial meta-ar alysis, then looked at the potential influence of the survey 
variables across levels of two moderator variables, year of publication and type of sample. 

All five studies considered some form of incentives, attempts to contact, postage, and sponsorship. 
Length and anonymity were considered in four of the five reviews, personalization and anonymity in three. 
Results were mixed for some of these variables, but it is important to look at the way in which the 
variables were operationalized by the various reviewers. 



Figure 1 

Significant effects 
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Other incentive 
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‘See article for complete listing of variables coded under this heading 
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Figure 2 



Small Significant, Situation Specific, or Mixed Results 
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Results 


Followup postcard 


Include reply 
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Colored paper 
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Anonymity 


First class outgoing 
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postage 


return mail 
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Incentives < $1.00 
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Incentives > $1.00 
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FollowujVrepeated 






contacts 





“Detailed list of variables can be ordered from ASIS/NAPS microfiche publications as footnoted in the article. 



Incentives 

There seems to be little question that including monetary incentives increase response rates, and 
some indication that nonmonetary incentives may also be effective. If it is within the power of the 
researcher to use an incentive, the questions are what kind of incentive to use md, if it is a monetary 
incentive, how much it should be. Some of the researchers adjusted the amount of the incentives to current 
dollar values. Yammarino, Skinner and Childs (1991) categorized ' '^rtives as 50 cents or less, 50 cents to 
one dollar, more than one dollar, and nonmonetary. Incentives from 50 cents to one dollar were not 
significantly related to response rate, and monetary incentives in the other two categories appeared to be 
related to particular populations. Nonmonetary incentives were not significantly related to response rate. 

Fox, Crask and Kim (1988) used four values of monetary incentive: 10 cents, 25 cents, 50 cents, and 
one dollar. Weighted least squares regression was used to calculate the amount of increase in response rate 
that might be expected with the various incentives from approximately 10% for the 10 cent incentive to 
31% for the one dollar incentive. They also adjusted the incentive amounts using the consumer price index 
but found it was not helpful and ended up using their original, unweighted values. 

Heberlein and Baumgartner (1978) found the highest response rate for $1.00 incentive on the first 
contact, followed by a 25 cent incentive, no incentive, and a non-monetary incentive. Promised incentives 
produced the lowest response rates. Heberlein and Baumgartner's regression model obtained a percentage of 
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Figure 3 

Variables Not Related or Not Used in Prediction 
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•Detailed list of variables can be ordered from ASIS/NAPS microfiche publications as footnoted in the article. 

increase in response rate due to incentives by multiplying 6.1 times the incentive category values: (0), less 
than $ .25 (1), $ .25 (2), $ .50 (3), and $1.00 (4). 

Bruvold and Comer (1988) coded incentives as monetary and other , using a yes/no (present/ absent) 
coding for each. Monetary incentives were expected to improve response rates by 54%, other types of 
incentives by 31%. Linsky (1975) concluded that both monetary and nonmonetary incentives produced 
higher response rates than no incentives, with 25 cent reward appearing to be more effective than lesser 
amounts. 

Attempts to Contact 

Another area in which there is agreement is that additional contacts will increase response rates. 
What form the contacts should take may be the appropriate question. Fox, Crask and Kim (1988) focused on 
two variables: prenotification by letter which was associated with a 7.7% increase in response rate, and 
postcard follow-up, associated with 3.5% effect. Yammarino, Skinner and Childs (1991) found preliminary 
notification and a category referred to as follow-ups/repeated contacts each significantly related to 
response rate, with preliminary notification having been most effective in research in more recent years 
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(28.5%). Follow-ups/repeated contacts were more effective with institutional groups (30.6%) than 
consumer groups. 

Heberlein and Baumgartner (1978) found the advance contact and higher numbers of total contacts 
(from one to six) and follow-ups (from none to five) increased response rates. In their ten-variable 
prediction model, the total number of contacts was multiplied by 7.4, and a special third contact (in 
categories by type of contact) by 8.6 to obtain the percentage of increased response. 

Bruvold and Comer (1988) coded advance contacts as either used or not used and included a second 
variable consisting of the number of follow-ups (from zero to four). The 13% increase for advance contacts 
was the lowest they reported, except for the year in which the study was conducted. The total number of 
contacts was multiplied by .28 to achieve the anticipated increase due to follow-ups. In addition, they 
coded the researcher's use of second, third, and fourth contacts, with second (58%) and third (59%) contacts 
having more influence on response rate than either of the previously mentioned variables. 

Linsky (1975) noted that precontacts improved response rates in all studies that he examined, as 
did postcard follow-up reminders. Reminder letters, however, were as effective as postcards. 

Postage 

Outgoing Postage. Four of the reviews considered the type of postage used on outgoing mail. There 
is some, but not conclusive, evidence that special mail may be superior to regular first class, and that first 
class, stamped and/ or special mail produce higher response rates than second, third, or bulk rate or metered 
postage. Fox, Crask and Kim (1991) used two measures of comparison: first class versus second, third, or 
bulk rate postage; and stamped versus metered postage. Effect sizes (1.8% and 0.9% respectively) indicated 
a small impact on response rate favoring use of first-class postage. 

Bruvold and Comer (1988) initially coded the use or absence of metered postage, regular and special 
stamps as independent.variables. No coefficient was produced for regular stamps, while metered postage 
yielded a negative coefficient (- .23), and special stamps a positive one (. 27). 

Heberlein and Baumgartner (1978) found no difference between regular mail and other types of first 
contact. There were some differences between regular mail and special mail for the second, third, and 
fourth contacts, but the nature and extent of the difference (from 1% to 9%) did not did not consistently favor 
special mail. 2 While there was significance for a special third contact, that major difference was between 
personal/phone and no third contact, with regular mail and special mail contacts producing very similar 
response rates closer to the personal/phone rate of 93.9% than the response rate when no third contact was 
attempted (50.6%). 

Linsky (1975) had only four studies on which to study this issue. The results were mixed, with one 
study showing special delivery and air mail producing higher response rates than first class, other studies 

2 The reader is referred the detailed list of variables and results of significance tests that can be ordered 
from AS1S/NAPS microfiche publications as footnoted in the article. 
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showing little or no difference between first and third class and between air mail and uird class postage. 
The most dramatic difference was the superiority of airmail special delivery mail over the use of 
government-franked envelopes. 

Return Postage. There was considerable variation in the way return postage was studied. As was 
true regarding outgoing postage, the results are not uanimous. Fox, Crask and Kim (1988) examined stamped 
return postage versus business reply envelopes and found a 6.2% increase in response rate for stamped 
postage. Hand stamped return mail produced higher returns than envelopes mailed with a postal permit in 
each study examined by Linsky. 

For Yammarino, Skinner and Childers (1991), there were two categories for return postage: stamped 
versus metered (which had more effect with institutional groups than consumer groups) and special 
delivery versus air mail, which was not significantly related to response rate. Heberlein and Baumgartner 
(1978) did not find a significant difference between stamps and other types of postage on return envelopes. 
Bruvold and Comer (1988) coded both incoming metered postage and the use of a special stamp on incoming 
postage, but neither was included in predicting response rates. 

Linsky (1975) concluded that stamped envelopes were also more effective than unstamped 
envelopes, and a combination of stamps produced higher returns than a single stamp. Both Linsky and 
Yammarino, Skinner and Childers (1991) found that inclusion of a return envelope significantly improved 
the response rate. 

Sponsorship 

Results relating to sponsorship are unclear, partially because the reviews did not study the same 
types of Sponsorship. Part of the problem is due to the statistical requirements for a minimum number of 
cases in each category. HeberHn and Baumgartner (1978) originally categorized sponsorship as university, 
government, private, public health, and other, although there were only a small number of studies in the 
public and other categories. Government sponsorship had the highest average response rate of the 
remaining groups. Yammarino, Skinner and Childers (1991) found a significant effect for sponsorship, but 
the meaning is unclear. The authors commented that they did not have enough data points to examine the 
effect of government sponsorship and had to combine groups within the institutional category. Fox, Crask 
and Kim (1991) found an effect size of 8.9% favoring university sponsorship. It does not appear that other 
forms of sponsorship were coded, although the text comments on the comparison between university and 
business sponsorship. 

Bruvold and Comer (1988) coded each of four types of sponsors (university, government, private, and 
research firms) as present and absent. They found all four types of sponsors had a negative impact on 
response rate, ranging from -25%for university to - 84% for government sponsorship, linsky (1975) found 
response rates from studies with government, commercial, and university sponsorship very close in a single 
study, but with gpvemment sponsorship producing a significantly higher response. In another survey. 
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higher returns were received when respondents thought the study was conducted by a university laboratory 
than by the American Cancer Society. 

Length 

The relationship between questionnaire lengthy which can be measured in more than one way, and 
response rate is unclear. Yammarino, Skinner and Childers (1991) coded questionnaire length as four pages 
or less and greater than four pages. They found that questionnaire length exceeding four pages in length 
had an adverse impact on response rates. Heberlein and Baumgartner (1978) defined length in two ways, 
actual number of questions and actual number of pages. They also coded the time required for questionnaire 
completion (less than 15 minutes, 15 to 29 minutes, 30 to 60 minutes, and more than one hour). None of the 
three variables had a significant effect on response rate except when salience and number of contacts were 
controlled, in which case additional questions reduced response rates. The number of pages was multiplied 
by -.44% to determine the anticipated effect on response rate in Heberlein and Baumgartner's final response 
rate prediction model. 

Bruvold and Comer (1988) also used the number of questionnaire pages to represent length, but this 
was not included and assigned a role in their prediction model. Linsky (1975) looked at comparison of long 
and short questionnaires and found a larger number of studies showing no difference than those that did. 
When differences were found, they did not always favor the short questionnaires. 

Personalization 

Mixed results were found in the three reviews that considered personalization. It is not clear how 
this was defined and coded in the Yammarino, Skinner and Childers (1991) study, but it did not produce a 
significant effect. Heberlein and Baumgartner (1978) categorized whether or not a personally typed 
address on the envelope was used but found the 10% average difference in response rate (favoring 
personalization) was not significant. Linsky (1975), considering studies in which the cover letters were 
hand signed and/or the respondent was addressed by name, reported mixed findings, with over half of the 
studies achieving higher response rates for personalized letters, and the others nearly balanced between no 
difference and higher response rates for non-personalized letters. Even at this early stage in the research 
on survey methods, the suggestion was made that personalization may be less effective in certain 
circumstances. 

Anonymity 

This was not significant in the findings of Yammarino, Skinner and Childers (1991), Bruvold and 
Comer (1988), and Heberlein and Baumgartner (1978). Findings were mixed in the Linsky (1975) review. 

Other Variables 

There are some other variables which were not examined in a majority of the reviews but for which 
results should be noted: 
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Salience - Found to be a factor by both Heberlein and Baumgartner (1978) and Bruvold and Comer 
(1988), the only two reviews that attempted to code and include it. The possibility of subjectivity in coding 
this variable calls for additional attention to reliability. 

School or Army Population - S trongly linked to response rate by both Heberlein and Baumgartner 
(1978) and Bruvold and Comer (1988). 

A ppeals - A significant effect was found by Yammarino, Skinner and Childers (1991), mixed results 
by Linsky (1975). 

Deadline - Both Fox, Crask and Kim (1988) and Yammarino, Skinner and Childers (1991) found 
inclusion of a deadline for responding did not improve response rates. 

Author Background - Marketing background was negatively related to response rate by both 
Heberlein and Baumgartner (1978) and Bruvold and Comer (1988) while survey background had the 
strongest relationship (Bruvold and Comer). 

Colored Paper - Investigated only by Fox, Crask, and Kim (1988) who found a small significant 

effect. 

Postscript - Use of a postscript encouraging respondents to participate on the cover letter did not 
increase response rates in the Fox, Crask, and Kim (1988) review. 

Both Bruvold and Comer (1988) and Heberlein and Baumgartner (1978) were more detailed in 
coding the type of population, nature of the sample and questionnaire characteristics than the other 
reviews. Individuals with a strong interest in these areas are referred to the those two reviews. 

Discussion 

There is considerable variation among the nine review articles broadly examining mail survey 
response rate facilitation techniques. These are nine separate, sometimes overlapping, reviews rather than 
one review replicated eight times. Procedures tend to be better documented in the quantitative reviews, 
although it is recognized that writing style must be geared to the style preferred by the journal to which a 
manuscript is submitted. The existing level of technological and statistical sophistication at the time at 
which a study is conducted must also be taken into account. More recent authors have benefited from the 
computer search capabilities available to them in locating source articles as well as in expanded statistical 
alternatives in processing their data. Quantitative procedures tend to obscure the somewhat contradictory 
findings of individual research studies while taking into account the additive effects of multiple 
variables. The inconsistency of variable definitions sometimes limits the extent to which findings across 
review studies can be compared . 

The list of 329 source studies cited in these nine reviews is not exhaustive. Each review included 
studies not included in others covering the same time span. Differing selection criteria and search 
procedures may account for some of this, rather than the apparent failure to capitalize on previous reviews. 
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Those authors who documented their search procedures each went beyond previous lists or citation lists in 
other reviews in their efforts to uncover additional research studies. The most thorough searches included 
computer searches, citation lists in identified articles, and manual searches of journals. 

While computer searches facilitate the location of articles, those predating the computer abstract 
databases are less easily found. Potter, Sharpe, Hendee, and Clark (1972), Berdie and Anderson (1974), 
and Pressley (1976) pioneered the compilation of annotated indexes in mail survey research studies and 
each was cited in at least one of the reviews. Indexes to Survey Methodology Literature (U. S. Bureau of 
the Census, 1974) might be helpful in locating early references. The Bibliography of Marketing Research 
Methofds (Dickinson, 1986) could be a valuable resource in identifying current as well as older articles. 
Dillman and Sangster (1990) cover the years 1974-1989 and provide an update for the extensive 
bibliography found in Dillman (1978). Public Opinion Quarterly maintains a cumulative index by topic to 
articles it publishes. 

The choice of keywords or descriptors used in computerized searches of abstract databases 
influences the results. Some of the reviewers may not have discovered relevant articles because the 
terminology used in their searches was too limited. Searches using "response rate" did not produce 
identical lists of citations to those when "response rates" was used, except in the ABI/Inform database 
which shortened the plural terms to the singular forms. 

Differences in databases used and their contents also influence the outcomes of the search efforts. 
PsycINFO is an online database of psychological publications that includes dissertations, which are not 
included in PSYCLit. The Social SciSearch database, used by some of the review authors, only permits 
searches of the titles. Social SciSearch is the online version of Social Sciences Citation Index, which is 
very inclusive. Having any type of computerized search process, though limited to title, is a distinct 
improvement over using the bound volumes of the Index. The databases that were searched in the present 
study (ERIC, ABI/Inform, PSYCLit, and Sociofile) were all available on CD-ROM and all contained 
abstracts as well as titles and descriptors or keywords. In searching the databases, the entire entry 
(including abstract) was searched for the search term(s). 

Fox, Crask, and Kim (1988) expressed concern that using only published studies might bias the 
results, because studies that failed to find significant effects might not appear in print. They did note, 
however, that published studies were easier to obtain. Because many studies have more than one variable 
being manipulated at a time, nonsignificant results for individual factors were more prevalent than 
significant ones in their data. Only 23 percent of the 214 effect estimates in their study were statistically 
significant. 

Few journals have been consistent publishers of mail survey response rate studies through the years. 
The number of journal articles found by the reviewers may, to some degree/ be related to the search 
procedures employed by the reviewers. Those journals searched most thoroughly (manually, issue by issue) 
had the largest numbers of citations in the overall list of 329 source citations. The ten journals manually 
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searched by Yammarino, Skinner, and Childers (1991), are the ten journals with the largest numbers of 
citations in the total list of source articles. Similarly, the three journals searched by Fox, Crask, and Kim 
(1988) were the three with the largest numbers of source article citations, and four of the five journals 
thoroughly searched by Bruvold and Comer (1988) are the most frequently cited journals in the total list. 

Publication of studies on mail surveys have been published primarily in journals with audiences in 
public opinion research and marketing. Sociological and psychological journals have also been open to such 
studies, but to a lesser extent. By broadly defining educational journals (Journal of Educational Research, 
Educational Research Bulletin, Journal of Educational Psychology, Research in Higher Education, 
Vocational Guidance Quarterly, Journal of Experimental Education, and College Student Journal), there was 
a total of 10 articles (3 percent) in the list of 329 source citations that can be attributed to any educational 
journals. It is small wonder that education was omitted from Dillman's (1991) statement that "Statistics, 
psychology, marketing research, economics, and the various health sciences are disciplines in which 
research efforts to improve mail survey methods are regularly conducted and reported" (p. 226). 

It is discouraging to note that no review articles were located in education journals through the 
ERIC search. Another observation that can be made at the conclusion of this study is that there was not a 
single article on the source citation list of 329 that was published in an American Educational Research 
Association (AERA) journal (i.e., American Educational Research Journal, Review of Educational Research, 
Journal of Educational Statistics, Educational Evaluation and Policy Analysis, Review of Research in 
Education). This is particularly disturbing to those who consider AERA as the foremost organization for 
researchers in the field of education. While AERA publishes the Journal of Educational Statistics, this 
publication does not provide a forum for studies on design issues relating to research on mail survey 
methods. Included in the 137 ERIC citations resulting from the computerized search were 20 papers on 
survey research methodology that had been presented at annual meetings of AERA. This finding supports 
the contention that the descriptors used in the search were relevant 

Some variables were studied more extensively (by more review authors) than others. It has been 
fairly well accepted that incentives and followups increase response rates, and this was confirmed by the 
reviews. Response rates are higher when incentives are used, but the optimal value and type of incentive is 
elusive. Response rates also increase as additional efforts are made to contact individuals. Special and/ or 
first class outgoing postage appears to be more conducive to responding than lesser forms of postage. There is 
some evidence that the effectiveness of different types of return postage may be related to the population 
being surveyed. 

Findings are inconclusive regarding sponsorship, questionnaire length, personalization, anonymity, 
and appeals as they relate to response rates. Some variables produced consistent findings but were included 
in only one or two reviews. Saliency, school or army populations, and colored paper appear to facilitate 
response rates while marketing background of the author is an inhibitor. Setting a deadline for responding 




16 



18 



I 



and including a postscript on the cover letter had no recognizable effect, either positive or negative, on 
response rate. 

As was pointed out by Bruvold and Comer (1988), some variables related to a particular survey are 
fixed, while others offer varying degrees of opportunity for manipulation. The sample source, population 
type, relationship of the respondent to the sponsor, and the year in which the survey is conducted are not 
subject to manipulation. It has been demonstrated that a response rate prediction model that is developed 
without respect to population may not be applicable for a specific population, such as those involved in 
industrial surveys (Jobber & Saunders, 1993). The effect of at least some mail survey procedures have also 
been shown to vary according to the year in which the survey was conducted as well as whether the 
population was a consumer or an institutional group (Yammarino, Skinner, & Childers, 1991). 

While much can be learned from review articles, this study calls attention to the disparity with 
which the procedures used in the integrated review articles were documented. Areas in which findings 
were inconsistent may be those most prone to situation specificity and should, perhaps, be the focus of future 
research rather than continued study of variables for which findings seem to generalize, such as incentives 
and repeated contacts. 
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